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DETAILED ACTION 

1 . The application has been examined. Claims 1-33 are pending in this office action. 



2. Response to communications filed on March 03, 2003 for claims 1 - 33 in this office 
action. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claims 1 , 2, 4, 5, 7, 9, 24, 25 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent 5,500,920 issued to Julian M. Kupiec (hereinafter 
"Kupiec") in view U.S. Patent 6,272,457 issued to Terry Edward Ford et al 
(hereinafter "Ford"). 

Regarding claim 1 , Kupiec teaches a method for providing collateral information 
for inclusion with an information stream, comprising steps of: 

examining (see Kupiec column 14, lines 15-20) the information stream to 
recognize a presence of events that occur in the information stream (see Ford column 
2, lines 34 -36). 
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automatically generating database queries (see Kupiec column 34, line 29) from 
recognized events (see Ford column 2, lines 29 - 33). 

analyzing database query results so as to rank and select database query results 
to be inserted into the information stream as collateral information (see Kupiec column 
2, lines 26 - 27 and column 12, lines 1 - 6). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec with the teaching of Ford wherein 
the system automatically generate the digital library from the audio/video data. This 
library could be examined, searched, desired data extracted and appended to an 
information stream. The motivation is that the interactive tools allow the user to review 
all speech entries and correct as required. 

Regarding claim 2, Kupiec teaches the step of analyzing comprises a step of 
ranking the database query results based on a plurality of criteria (see column 12, lines 
21 - 22). 

Regarding claim 4, Kupiec teaches the database queries are automatically 
generated (see column 34, line 29) based on information corresponding to a list that 
identifies topics in text that is automatically extracted from the information stream (see 
column 2, lines 45 - 52), where the topics correspond to predetermined topic 
taxonomies (see column 12, lines 34 - 40). 
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Regarding claim 5, Kupiec the step of examining comprises a step of 
automatically extracting text from the information stream (see column 2, lines 45 - 52). 

Regarding claim 7, Kupiec teaches the step of automatically extracting text from 
the information stream (see column 2, lines 45 - 52) comprises a step of operating a 
voice recognition system (see Ford column 5, lines 8-9) 

Regarding claim 9, Kupiec teaches the step of automatically extracting text from 
the information stream comprises a step of operating a character recognition system 
(see column 24, lines 32 - 35). 

Regarding claim 24, Kupiec teaches a computer-implemented method for generating 
collateral information from an audio/video stream comprising steps of: 

examining (see Kupiec column 14, lines 15-20) the audio/video stream to 
recognize a presence of events that occur in the audio/video stream (see Ford column 
2, lines 34 -36). 

generating database queries (see Kupiec column 34, line 29) from recognized 
events (see Ford column 2, lines 29 - 33), and 

analyzing database query results so as to rank and select database query results 
to be presented as the collateral information (see Kupiec column 2, lines 26 - 27 and 
column 12, lines 1 -6). 
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It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec with the teaching of Ford wherein 
the system automatically generate the digital library from the audio/video data. This 
library could be examined, searched, desired data extracted and appended to an 
information stream. The motivation is that the interactive tools allow the user to review 
all speech entries and correct as required. 

Regarding claim 25, Kupiec teaches wherein further comprising a step of 
inserting the collateral information into the audio/video stream in real time or 
substantially real time (see column 21 , lines 23 - 26). 

5. Claims 12, 13, 14, 15, 16, 17 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kupiec in view of U.S. Patent 5,835,667 issued to Howard D. 
Wactlar et al (hereinafter "Wactlar"). 

Regarding claim 12, Kupiec teaches a method for providing collateral information 
for multiplexing with an information stream, comprising steps of: 

Kupiec teaches automatically generating queries from the information elements 
(column 34, line 29) for searching at least one database (see column4, lines 58). 

Kupiec does not explicitly teach converting the information stream into text. 

Analyzing the text to identify information elements, 
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extracting data from database search results that is relevant to the information 
stream; and multiplexing the data into the information stream for presentation at a 
destination of the information stream. 

However, Wactlar teaches converting the information stream into text (see 
column 17, lines 43 - 45); 

analyzing the text to identify information elements (see column 9, lines 42 - 43); 

extracting data from database search results that is relevant to the information 
stream (see column 15, lines 2 - 6); and 

multiplexing the data into the information stream for presentation at a destination 
of the information stream (see column 15, lines 7 - 26). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec with teaching of Wactlar wherein 
tentative entry is created in the database. The motivation being that it facilitates the 
storing of characterizing information if the user then corrects or changes the text using 
voice commands. 

Regarding claim 13, Wactlar the step of extracting comprises a step of ranking 
the extracted information based on a plurality of criteria, and where the step of 
multiplexing uses the ranked data (see column 15, lines 7 - 26). 

Regarding claim 14, Kupiec teaches the step of extracting comprises a step of 
ranking extracted document information based on a score derived from a free text 
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search of a document database using the text, on a number of named entities extracted 
from the text that are found in the documents (see column 2, lines 45 - 52 and column 
4, lines 66 - 67) 

Kupiec does not explicitly teach a taxonomy path score, where the taxonomy 
path score represents an amount of relatedness between a taxonomy-related 
information element identified in the text and a predetermined taxonomy tree. 

However Wactlar teaches a taxonomy path score, where the taxonomy path 
score represents an amount of relatedness between a taxonomy-related information 
element identified in the text and a predetermined taxonomy tree (see column 13, lines 
13-15). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine the teaching of Kupiec with the teaching of Wactlar 
wherein steps of ranking extracted document information is based on a score and aging 
factor. Consequently, an event is less likely to be considered as time passes. 

Regarding claim 15, Wactlar teaches the queries are generated based on 
information elements that correspond to a list of information elements identifying topics 
in the text being analyzed, where the topics correspond to predetermined topic 
taxonomies (see column 6, line 67 and column 7, lines 1 - 10). 

Regarding claim 16, Wactlar teaches the step of analyzing the text comprises 
steps of segmenting the text into sentences and a step of operating on the sentences to 
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identify topics that correspond to predetermined topic taxonomies, and wherein the step 
of automatically generating queries operates on identified topics (see column 6, line 67 
and column 7, lines 1-10). 

Regarding claim 17, Wactlar teaches the step of analyzing the text comprises 
steps of at least segmenting the text into sentences (see column 6, line 67 and column 
7, lines 1-10), identifying names of entities within the text, and a step of operating on 
the sentences to identify topics that correspond to predetermined topic taxonomies (see 
column 17, lines 48 - 51), and wherein the step of automatically generating queries 
operates on identified topics and ranks the database search results based at least on 
numbers of named entities found and on an amount of relatedness between a 
taxonomy-related information element identified in the text and a predetermined 
taxonomy tree (see column 13, lines 13-15). 

6. Claim 18 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kupiec in 
view of U.S. Patent 5,970,460 issued to Eric Bunce et al (hereinafter "Bunce"). 

Regarding claim 18, Kupiec teaches a system for providing collateral information for 
inclusion with an information stream, said system operating in real time or substantially 
real time and comprising: 

automatically generating database queries from recognized events (see column 
34, lines 29). 
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a subsystem, having an input coupled to an output of said database (see column 
2, lines 24 - 30), for analyzing database query results so as to rank and select database 
query results to be inserted into the information stream as collateral information (see 
column 2, lines 26 - 27 and column 12, lines 1 - 6), 

a database for receiving said database queries (see column 2, lines 24 - 30) 

Kupiec does not explicitly teach a subsystem for examining the information 
stream to recognize a presence of events that occur in the information stream; 

a subsystem, having an input coupled to an output of said examination 
subsystem, for automatically generating database queries from recognized events. 

However, Bunce teaches a subsystem for examining the information stream to 
recognize a presence of events that occur in the information stream (see column 1 , 
lines 63 - 57 and column 2, lines 1 - 3); 

a subsystem, having an input coupled to an output of said examination 
subsystem (see column 1 , lines 63 - 67 and column 2, lines 1 - 3). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec with teaching of Bunce wherein the 
a system for providing collateral information for inclusion with an information stream is 
attained. The motivation is that the process also produces speech/audio event 
database, which can be used as an index into diction event database for commands 
correcting previously misrecognized diction event. 
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7. Claims 3, 6, 8, 10, 11, 26, 27, 31, 32 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kupiec in view of Ford and further in view of Wactlar. 

Regarding claim 3, Kupiec teaches the plurality of criteria comprise a score 

derived from a free text search of the database 
(see column 4, lines 66 - 67) using text that is automatically extracted from the 
information stream, on a number of named entities appearing in the text and in the 
database query results (see column 2, lines 45 - 52). 

Kupiec or Ford do not explicitly teach and on a taxonomy path score, where the 
taxonomy path score represents an amount of relatedness between a taxonomy-related 
information element found in the text and a predetermined taxonomy tree. 

However, Wactlar teaches and on a taxonomy path score, where the taxonomy 
path score represents an amount of relatedness between a taxonomy-related 
information element found in the text and a predetermined taxonomy tree (see column 
13, lines 13-15). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec and Ford with the teaching of 
Wactlar wherein the object are classified. The motivation is that segmenting video by 
appearance of a particular object or a combination is a powerful tool to detect a 
particular class of object. 
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Regarding claim 6, Kupiec teaches wherein the step of automatically generating 
database queries operates on identified topics (see column 34, line 29) 

Kupiec or Ford do not explicitly teach segmenting the text into sentences; and 
operating on the sentences to identify topics that correspond to predetermined topic 
taxonomies 

However, Wactlar teaches segmenting the text into sentences; and operating on 
the sentences to identify topics that correspond to predetermined topic taxonomies (see 
column 3, lines 31 - 38. column 6, line 67 and column 7, lines 1-10). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec and Ford with the teaching of 
Wactlar wherein the system automatically generate the digital library from the 
audio/video data. This library could be examined, searched, desired data extracted and 
appended to an information stream. The motivation is that the interactive tools allow the 
user to review all speech entries and correct as required. 

Regarding claim 8, Kupiec teaches the step of automatically extracting text from 
the information stream (see column 2, lines 45 - 52) 

Kupiec or Ford does not explicitly teach comprises a step of extracting closed 
caption text. 

However, Wactlar teaches comprises a step of extracting closed caption text 
(see column 7, lines 58 - 62) 
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It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching or Kupiec and Ford with the teaching of 
Wactlar wherein to induct the segmenter goes through the process of extracting text 
before returning errors on the closed-caption data. The motivation being that since 
close-caption data contains punctuation mark, it makes it possible to introduce sentence 
break in the usual way. 

Regarding claim 10, Wactlar teaches the step of automatically extracting text 
from the information stream comprises a step of also generating text that is descriptive 
of a number of human faces that are present in an image conveyed by the information 
stream (see FIGs. A-1 and A-2). 

Regarding claim 1 1 , Wactlar teaches the step of examining the information 
stream further comprises steps of segmenting the text into sentences (see column 7, 
lines 26 - 27) and a step of operating on the sentences to identify topics that 
correspond to predetermined topic taxonomies and the presence of names of entities 
(see column 6, line 67 and column 7, lines 1 - 10), and further comprising steps of 
assembling a list comprised of an identified topic having a start time and an end time, as 
well as any named entities that occur between the start time and the end time, 
assembling a query object comprised of named entities that occur between the start 
time and the end time of the identified topic (see column 19, lines 28 - 30), searching at 
least one database to identify a first set of stored documents that correspond to the 
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topic, identifying a subset of the first set of documents that contain the named entities, 
identifying a second set of documents that correspond to words found in the text (see 
column 7, lines 28 - 33); scoring the returned documents based on a plurality of criteria 
and ranking the documents based on their scores (see page 10, lines 19 - 22). 

Regarding claim 26, Kupiec and Ford does not explicitly teach the step of 
examining includes a step of generating a speech transcript from at least the audio 
portion of the audio/video stream, and wherein recognized events comprise speech 
topics 

Wattle t eaches the step of examining includes a step of generating a speech 
transcript from at least the audio portion of the audio/visual stream, and wherein 
recognized events comprise speech topics (see column 6, lines 41 - 48). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching or Kupiec and Ford with the teaching of 
Wactlar wherein the audio data is subjected to the function of speech and language 
interpretation. The motivation being that the digital video library is made available to 
users that desire it. 

Regarding claim 27, Wactlar teaches the audio/video stream originates as a 
television broadcast signal (see column 1 , lines 25 - 33). 
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Regarding claim 31 , Kupiec teaches the database queries are automatically 
generated (see column 34, lines 29). 

Kupiec and Ford does not explicitly teach based on information corresponding to 
identified topics extracted from the audio/video stream, where the topics correspond to 
predetermined topic taxonomies. 

However, Wactlar teaches based on information corresponding to identified 
topics extracted from the audio/video stream, where the topics correspond to 
predetermined topic taxonomies (see column 6, lines 41 - 48 and column 1 3, lines 13 - 
15). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching or Kupiec and Ford with the teaching of 
Wactlar wherein database queries are automatically generated. The motivation is that 
the process also produces speech/audio event database, which can be used as an 
index into diction event database for commands correcting previously misrecognized 
diction event. 

Regarding claim 32, Wactlar the step of examining includes steps of generating a 
speech transcript comprised of words from at least the audio portion of the audio/video 
stream (see column 2, lines 44 - 53); segmenting the words into sentences; and 
operating on the sentences to identify topics that correspond to predetermined topic 
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taxonomies, wherein the step of generating database queries operates on identified 
topics (see column 6, line 67; column 7, lines 1-10 and column 19, lines 28 - 30). 

8. Claims 19-23 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Kupiec in view of Bunce and further in view of Wactlar. 

Regarding claim 19, Kupiec teaches the analyzing subsystem employs ranking 
criteria comprised of a score derived from a free text search of the database using text 
that is automatically extracted from the information stream, on a number of named 
entities appearing in the text and in the database query results (see column 2, lines 45 - 
52). 

Kupiec or Bunce does not explicitly teach and on a taxonomy path score, where 
the taxonomy path score represents an amount of relatedness between a 
taxonomy-related information element found in the text and a predetermined taxonomy 
tree, and wherein the query generation subsystem generates queries based on 
information corresponding to a list that identifies topics in the text that is automatically 
extracted from the information stream, where the topics correspond to elements of the 
taxonomy tree. 

However, Wactlar teaches and on a taxonomy path score, where the taxonomy 
path score represents an amount of relatedness between a taxonomy-related 
information element found in the text and a predetermined taxonomy tree, and wherein 
the query generation subsystem generates queries based on information corresponding 
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to a list that identifies topics in the text that is automatically extracted from the 
information stream, where the topics correspond to elements of the taxonomy tree (see 
column 13, lines 13-15). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec and Bunce with the teaching of 
Wactlar wherein the system automatically generate the digital library from the 
audio/video data. This library could be examined, searched, desired data extracted and 
appended to an information stream. The motivation is that the interactive tools allow the 
user to review all speech entries and correct as required. 

Regarding claim 20, Kupiec said examining subsystem comprises at least one 
unit for automatically extracting text from the information stream (see column 2, lines 45 



Kupiec or Bunce does not explicitly teach a unit for segmenting the text into 
sentences and at least one unit for operating on the sentences to identify topics that 
correspond to predetermined topic taxonomies, wherein said query generation 
subsystem automatically generates database queries based at least in part on identified 
topics. 

However, Wactlar teaches a unit for segmenting the text into sentences and at 
least one unit for operating on the sentences to identify topics that correspond to 
predetermined topic taxonomies, wherein said query generation subsystem 



-52). 



Application/Control Number: 09/698,894 Page 17 

Art Unit: 2172 

automatically generates database queries based at least in part on identified topics (see 
column 6, line 67; column 7, lines 1-10). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec and Bunce with the teaching of 
Wactlar wherein the system automatically generate the digital library from the 
audio/video data. This library could be examined, searched, desired data extracted and 
appended to an information stream. The motivation is that the interactive tools allow the 
user to review all speech entries and correct as required. 

Regarding claim 21 , Kupiec or Bunce does not explicitly teach said text 
extracting unit comprises at least one of a voice recognition system, a system for 
extracting closed caption text, and a character recognition system. 

However, Wactlar teaches said text extracting unit comprises at least one of a 
voice recognition system, a system for extracting closed caption text, and a character 
recognition system (see column 6, lines 39 - 40 and column 7, lines 58 - 62). 

Regarding claim 22, Kupiec or Bunce does not explicitly teach said examining 
subsystem comprises a unit for generating text that is descriptive of a number of human 
faces that are present in an image conveyed by the information stream. 

However, Wactlar teaches said examining subsystem comprises a unit for 
generating text that is descriptive of a number of human faces that are present in an 
image conveyed by the information stream (see FIGs. A-1 and A-2). 
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Regarding claim 23, Kupiec or Bunce does not explicitly teach comprising a unit 
for operating on the sentences to identify the presence of names of entities, and further 
comprising a unit for assembling a list comprised of an identified topic having a start 
time and an end time, as well as any named entities that occur between the start time 
and the end time, and where the query generation subsystem assembles a query object 
comprised of named entities that occur between the start time and the end time of the 
identified topic for searching said database to identify a first set of stored documents 
that correspond to the topic, a subset of the first set of documents that contain the 
named entities, a second set of documents that correspond to words found in the text; 
and where said analyzing subsystem scores the returned documents based on a 
plurality of criteria and ranks the documents based on their scores. 

However, Wactlar teaches comprising a unit for operating on the sentences to 
identify the presence of names of entities, and further comprising a unit for assembling 
a list comprised of an identified topic having a start time and an end time, as well as any 
named entities that occur between the start time and the end time (see column 19, lines 
28 - 30), and where the query generation subsystem assembles a query object 
comprised of named entities that occur between the start time and the end time of the 
identified topic for searching said database to identify a first set of stored documents 
that correspond to the topic, a subset of the first set of documents that contain the 
named entities (see column 7, lines 28 - 33), a second set of documents that 
correspond to words found in the text; and where said analyzing subsystem scores the 
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returned documents based on a plurality of criteria and ranks the documents based on 
their scores (see column 9, lines 42 - 55). 

9. Claims 28, 29, 30 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Kupiec in view of Ford and further in view of U.S. Patent: 5,786,814 issued Thomas 
P. Moran (hereinafter "Moran"). 

Regarding claim 28, Kupiec or Ford does not explicitly teach the audio/video 
stream originates at a meeting, and further comprising a step of presenting the collateral 
information to meeting participants in real time or substantially real time 

Moran teaches the audio/video stream originates at a meeting, and further 
comprising a step of presenting the collateral information to meeting participants in real 
time or substantially real time (see column 5, lines 25 - 60). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine teaching of Kupiec and Ford with the teaching of Moran 
wherein indices are created into meeting recording. The motivation is that this provides 
an intuitive way for people to gain direct access to portions of the recorded session. 

Regarding claim 29, Moran teaches the step of presenting comprises a step of 
inserting the collateral information into the audio/video stream, and displaying the 
audio/video stream to the meeting participants (see column 29, line 67 and column 30, 
lines 1-9). 
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Regarding claim 30, Moran a step of archiving at least the collateral information 
(see column 30, lines 10-18). 

10. Claim 33 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kupiec. 

Regarding claim 33, Kupiec teaches a computer readable media having recorded 
thereon a program for providing collateral information for inclusion with an information 
stream (see column 21 , lines 23 - 26), the program comprising instructions for 
examining the information stream to recognize a presence of events that occur in the 
information stream (see column 14 lines 15-20), for automatically generating database 
queries from recognized events (see column 34, line 29); and for analyzing database 
query results so as to rank and select database query results to be inserted into the 
information stream as collateral information (see column 2, lines 26 - 27 and column 
12, lines 1 -6). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to modify teaching of Kupiec wherein the system automatically 
generate the digital library from the audio/video data. This library could be examined, 
searched, desired data extracted and appended to an information stream. The 
motivation is that the interactive tools allow the user to review all speech entries and 
correct as required and append the result to the stream of information. 
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Conclusion! 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Fred I. Ehichioya whose telephone number is 703-305- 
8039. The examiner can normally be reached on M - F 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kim Y. Vu can be reached on 703-305-4393. The fax phone numbers for 
the organization where this application or proceeding is assigned are 703-746-7239 for 
regular communications and 703-746-7238 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is 703-303- 
3900. 

Fred Ehichioya i] / 



May 17, 2003 
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